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Abstract: This article is an outgrowth of the evaluation and assessment process in a large 
interdisciplinary studies (IDS) program at West Virginia University. We argue for the im¬ 
portance of collecting and assessing quantitative and qualitative data concerning learning 
outcomes, student satisfaction, career choice and development, and graduate education after 
students have graduated from an undergraduate IDS program. While there are difficulties 
inherent in developing an assessment instrument and gathering quality data, we contend that 
this process ultimately yields valuable insights that can improve student learning, teacher 
quality, and programmatic design. Moreover, a comprehensive postgraduate tool can provide 
evaluative data directly relevant to critics’ demands for higher education to implement greater 
accountability measures and demonstrate programmatic achievement. To assist other IDS 
programs, this article includes a practical guide to navigating the difficulties in developing a 
postgraduation evaluation and assessment tool and carrying out a plan for collecting data on 
a large scale. 

Keywords : interdisciplinary, assessment, evaluation, postgraduate, capstone, accreditation, 
multidisciplinary 

For at least the past decade ferocious competition and widespread criti¬ 
cism of spiraling tuition costs have prodded American universities into con- 
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certed efforts to improve the academic experience of their undergraduate 
students. Areas of emphasis include improving student learning and stu¬ 
dent-community engagement, developing stronger critical thinking skills, 
enhancing the quality of instruction, and providing clear pathways to suc¬ 
cess in postgraduation life. To determine whether such efforts are proving 
successful, universities are mandating an increase in both the number and 
sophistication of evaluation and assessment measures. Meaningful evalu¬ 
ation and assessment require considerable contact with students at regular 
stages in their academic curriculum, so the burden of implementing new 
evaluation and assessment practices has fallen in great part on the programs 
and departments that comprise the academic backbone of the university 
system. This article will examine a specific response to the demand for in¬ 
creased programmatic accountability: the use of a postgraduation research 
instrument to facilitate evaluation and assessment of an interdisciplinary 
program. 

Evaluation and assessment measures are continually evolving. In only a 
few decades, the standard for good academic practice has evolved from spo¬ 
radic course-based assessments of student learning to advanced quantitative 
and qualitative metrics illuminating the academic experience of students at 
the course, programmatic, and institutional levels. For example, peer review 
among professors, the development of teaching resource centers, and the 
creation of “lead teacher” programs are providing new data on teacher qual¬ 
ity (Field & Stowe, 2002). Moreover, expanded cross-disciplinary use of 
authentic assessment tools like rubrics and more systemic efforts to increase 
experiential learning opportunities such as field service placements, co-ops, 
and lab time have increased the data available on analytical and critical 
thinking skill development, research quality and accessibility, and student- 
community engagement (Field & Stowe, 2002). 

While we applaud each of these new measures, we also argue that there is 
another valuable source of data currently underutilized by institutions and, 
more specifically, interdisciplinary studies programs: postgraduation assess¬ 
ment. Currently, most postgraduation research is driven by a desire to com¬ 
pile information quantifying the career and financial successes of university 
graduates and thus validate the value of a university degree. Although this 
evaluative information is important, particularly to interdisciplinary and lib¬ 
eral arts programs that lack clearly defined career paths, postgraduation as¬ 
sessment has potential that exceeds basic degree evaluation. When carried 
out correctly, comprehensive postgraduation assessment not only facilitates 
the evaluation of a program’s worth; it also becomes a means to advance 
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the entire academic assessment project, encouraging graduates to reflect on 
learning outcomes, teacher quality, student satisfaction, and graduate educa¬ 
tion choices, and then relate these considerations to their current lives, both 
inside and outside of the workplace. 

In this article we present a practical rationale for and guide to conducting 
postgraduation evaluation and assessment in an interdisciplinary program. 
We initiate the discussion by presenting the central arguments for conduct¬ 
ing programmatic evaluation and assessment, including meeting higher ed¬ 
ucation accountability requirements and examining learning outcomes. Ad¬ 
ditionally, programmatic assessment is discussed for its applicability to all 
academic programs and then specifically related to the unique challenges 
facing interdisciplinary studies programs. We then articulate our rationale 
for conducting postgraduation evaluation and assessment as a means of col¬ 
lecting valuable data that buttress other forms of programmatic evaluation 
and assessment. The final section presents a practical guide to the practice 
of postgraduate evaluation and assessment in an interdisciplinary studies 
program by relating our experiences in carrying out the process and pro¬ 
viding a demonstration of how we used our research tool to improve our 
program. 

Reasons for Programmatic Evaluation and Assessment 

Before addressing postgraduation evaluation and assessment, we should 
clarify how programmatic assessment differs from other types of student- 
focused, evaluative research and why interdisciplinary studies programs 
should conduct programmatic evaluation and assessment. Such a discussion 
must begin with a clarification of the terms “assessment” and “evaluation.” 

Assessment and evaluation represent two related, but ultimately distinct 
research agendas. Stowe and Eder (2002) have delineated the division: 

In general, when data are collected to improve teaching and learning, 
the process is assessment. . . When data are collected to make judg¬ 
ments of worth or value, the process is evaluation. Judgments might 
involve qualification for tenure, continuation of a program, the level 
of funding, or continuation of accreditation, (p. 81) 

When the terms are defined side-by-side, the distinction is clear: Assess¬ 
ment is process-based and evaluation is summative—though in practice the 
line between the two may blur. Evaluation demonstrates if a person, pro¬ 
gram, or institution is succeeding or failing, but cannot alone explain the cir- 



78 


79 


Benjamin Brooks & Evan Widders 


cumstances leading to success or failure or, perhaps more importantly, mea¬ 
sures for improvement. Only comprehensive assessment can fill this void. 

Stowe and Eder (2002) acknowledge, however, that many in academia 
confuse the two concepts, and our experience at West Virginia University 
supports their claim. At West Virginia University, as at the vast majority 
of institutions for higher education, programmatic assessment is mandatory 
(Kuh & Ewell, 2010). Department administrators and faculty do not, how¬ 
ever, uniformly share the institutional enthusiasm for assessment, and so the 
spectrum of assessment practice is wide. Some programs have embraced 
yearly assessment as a means to illuminate opaque concepts such as aca¬ 
demic accomplishment related to learning outcomes. Other programs, per¬ 
haps resenting the mandate from above to implement sweeping measures, 
view assessment as a burdensome and unwelcome intrusion into their af¬ 
fairs by higher administration and decline to fully engage with its practice. 
Although they may claim to be practicing assessment, reluctant programs 
are most often simply evaluating, often using minimal demographic and re¬ 
tention/graduation statistics to argue to their stakeholders for the continued 
relevance of their programs (Miller, 2006; Basken, 2007). 

We acknowledge that comprehensive assessment is certainly much more 
time-consuming and labor-intensive than collecting evaluative data, and that 
assessment research is oftentimes slighted by promotion and tenure commit¬ 
tees, but it is also a mistake for academic programs to fail to embrace it fully. 
There are at least two clear reasons to adopt thorough assessment practices. 
First, the institutional mandates being handed down to programs actually 
stem from changes in higher education policy calling for educational ac¬ 
countability, and these policies continue to gather strength with each round 
of yearly tuition hikes (see for example. The National Governors Associa¬ 
tion 2010 report “Complete to Compete”) to the point where basic evalua¬ 
tive data are no longer sufficient to meet the demands of governing bodies. 
Secondly, a well-reasoned assessment plan with an emphasis on learning 
outcomes facilitates concrete improvements in both student learning and in¬ 
structional quality. 

Higher Education Accountability 

For at least the last few decades, federal and state initiatives have urged 
universities to accept accountability for their educational practices. This 
policy decision stems from a 1990 Department of Education mandate re¬ 
quiring higher education accrediting organizations to include documenta¬ 
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tion addressing student learning when reviewing institutions (Ewell, 2010). 
The focus on accountability has continued and intensified in recent years, 
through federal initiatives like the Spellings’ Commission on Fligher Educa¬ 
tion, which called for universities to be more transparent in demonstrating 
their academic effectiveness (Ekman & Pelletier, 2008). Practically, these 
mandates have led nearly every academic program to justify its current con¬ 
figuration, if not its continued existence, on a regular basis. 

While the necessity to “sing for one’s supper” should be a relatively egali¬ 
tarian requirement across the university, the risks are often greater for newer 
programs or those viewed as non-traditional (Wilson, 2009). It is nearly im¬ 
possible to imagine a university shuttering a dysfunctional English program, 
for instance, though a change of emphasis from post-colonial literature to 
technical and business writing might be recommended. Lacking the tradi¬ 
tion, institutional advocacy, or tenured faculty of other, deeper-rooted pro¬ 
grams, interdisciplinary studies programs may be at greater risk. Moreover, 
recent examples, such as that at SUNY Albany, demonstrate that even tradi¬ 
tional major programs such as classics, French, and theater are in danger of 
extinction if they cannot demonstrate their continued relevance to students 
and the university’s mission (Flebel, 2010). Given the risks involved, adopt¬ 
ing a comprehensive evaluation and assessment plan is the most effective 
means to validate an interdisciplinary studies program’s merit to the univer¬ 
sity and its partners in accreditation. 

A common and accessible form of programmatic analysis is quantita¬ 
tive evaluation of basic student data—encompassing enrollment informa¬ 
tion, demographics and graduation/retention rates—gleaned from university 
databases. For example, the West Virginia Higher Education Policy Com¬ 
mission and the University Board of Governors mandate a program review 
every five years (West Virginia University Board of Governors, 2010). The 
instructions direct each university program to conduct a “self-study,” calling 
specifically for academic elements such as curriculum, course descriptions 
and yearly departmental assessment reports (discussed below). The ma¬ 
jority of the requested information, however, is aligned along quantitative 
measures of program vitality, including detailed reports of five-year enroll¬ 
ment trends, physical space and funding information, and estimates of the 
program’s ability to attract students and maintain a “viable, cost-effective 
program” (West Virginia University Board of Governors, 2010). Unlike the 
data provided by more thorough-going assessment practices, evaluative data 
are relatively simple to request from the Registrar or Admissions Office and 
may not require regular faculty participation to compile. Such evaluative 
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data-mining may seem particularly attractive to interdisciplinary programs. 
The discipline-focused structure of universities and the difficulties inher¬ 
ent in explaining the interdisciplinary enterprise to the uninitiated can act 
to discourage sophisticated assessment formulations (Stowe & Elder, 2002; 
Klein, 2002). Thus, for reasons of convenience, relying on basic student 
data to evaluate programmatic performance may be a tempting expedient for 
interdisciplinary programs. 

Relying on such a limited data set is perilous, however. In many cases, 
basic student data are either misleading or not sufficiently nuanced to dem¬ 
onstrate the true effectiveness of an interdisciplinary program. One way of 
mitigating this problem is to collect additional evaluative data—an opportu¬ 
nity we discuss below. To create a more complete picture of programmatic 
effectiveness, however, a comprehensive approach to assessment incorpo¬ 
rating learning outcomes is necessary. 

Learning Outcomes 

Learning outcomes represent the set of skills, knowledge, and/or be¬ 
haviors that a student is expected to have gained after instruction. 1 Due to 
the accountability measures described above, many academic units have 
adopted clearly defined learning outcomes that seek to calibrate student 
academic progress at different stages in the curriculum. Regularly collected 
learning outcome data provide academic programs with a descriptive sense 
of how well they are achieving the core academic values of their program 
and prescriptive ideas for improving individual assignments, courses, pro¬ 
grammatic mission, and teacher instruction. 

The descriptive data that are attained through learning outcome as¬ 
sessment can be used to demonstrate a program’s viability to accreditors 
through the process of yearly assessment reports. Regular assessment re¬ 
ports utilizing programmatic learning outcomes are a practical means to 
ensure that a program can clearly demonstrate its relevance to the universi¬ 
ty’s educational mission. Consistently compiled reports can also profitably 
supplement basic student data by documenting the program’s academic 
performance on a year-to-year basis. In doing so, assessment reports can 
provide a valuable counterpoint to raw evaluative data that are oftentimes 
outside of the program’s control. For instance, a change in financial aid 
eligibility determination formulae may lead to a drop in non-traditional 

1 Instructional Assessment Resources (IAR). (2007). Glossary, http://www.utexas. 
edu/academic/ctl/assessment/iar/glossary.php 
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student enrollment, raising questions about a program’s ongoing “viabil¬ 
ity” even though there was no decline in the program’s academic merit. 
Learning outcome-based assessment can then be brought to bear, mitigat¬ 
ing misjudgments based on such unavoidable variances in student enroll¬ 
ment data. 

Moreover, it is important to note that accreditation bodies are greatly in¬ 
terested in learning outcome assessment. For example. The North Central 
Association of Colleges and Schools, one of six institutional accreditors in 
the United States, enumerates five broad criteria for accreditation. The first 
two criteria are institutional in focus, but the remaining three are entitled 
“Student Learning and Effective Teaching,” “Acquisition, Discovery, and 
Application of Knowledge,” and “Engagement and Service.” 2 Providing ac¬ 
creditors with convincing evidence of accomplishment in these core crite¬ 
ria is difficult, if not impossible, without setting concrete learning outcome 
goals and improving the program in response to feedback. 

Prescriptive insights gained through learning outcomes assessment can 
be focused toward specific programmatic improvements. There are a num¬ 
ber of research studies that have been conducted that show the value of 
assessing learning outcomes as they relate to everything from capstone 
course development (see Brooks, Benton-Kupper, & Slayton, 2004) to 
learning environments (Wilson & Scalise, 2006). Unfortunately, assessing 
learning in an interdisciplinary studies program can be more problematic. 
As Boix Mansilla (2005), Newell (1994, 2001, 2006), and Repko (2007, 
2008a, 2008b), among others, have convincingly shown, disciplinary 
grounding is vital to the success of interdisciplinary programs and the in¬ 
tegrative thinking they promote. However, because most interdisciplinary 
programs rely on other academic units to provide disciplinary instruction, 
opportunities to convey and assess progress towards specifically interdis¬ 
ciplinary learning outcomes, like integrative thinking, may be limited. It is 
vital, then, to make good use of whatever interdisciplinary studies courses 
are incorporated into the curriculum by setting specifically interdisciplin¬ 
ary learning outcomes for students. Progress towards the desired outcomes 
can be then assessed by using comprehensive rubrics to evaluate assign¬ 
ments, encouraging students to complete surveys—at different stages of 
their studies if possible, and compiling feedback on instructor performance 

2 Higher Learning Commission (HLC). (2011). Institutional Accreditation: An Over¬ 
view, http://www.ncahlc.org/Infonnation-for-Institutions/criteria-for-accreditation. 
html 
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(Field & Stowe, 2002; Repko, 2008b). These data can then provide evi¬ 
dence for academic success as well as revealing a path for positive changes 
to the curriculum. 

Still, given the difficulties created by intermittent academic interaction 
with students in many interdisciplinary programs, more creative means of 
data collection may be required. As we will see, postgraduation assessment 
can be a powerful tool to supplement more traditional, classroom-centered, 
assessment methods. 

Postgraduation Evaluation and Assessment 

Postgraduation evaluation and assessment require the collection of data 
related to a graduated student’s academic experience. Admittedly, this is 
not a new tradition in American higher education. As far back as the 1930s 
institutions of higher education, state agencies, and for-profit businesses 
like the College Board began collecting data on students after graduation, 
usually in the form of alumni surveys (Ewell, 2005). These surveys gener¬ 
ally served as a method for institutions to evaluate overall alumni satisfac¬ 
tion with university services, while also keeping a watchful eye on potential 
fund-raising sources. Perhaps of greater importance at the programmatic 
level, these data allowed programs to evaluate the real-world value of the 
degrees they conferred on their graduates. Assessment, however, was not 
yet a priority. It is only within the last few decades, with the rise of interest 
in higher education accountability, that institutions have begun to query 
alumni about elements of their academic experience, such as learning out¬ 
comes, that are more comprehensively addressed by assessment than evalu¬ 
ation (Ewell, 2005). 

Unfortunately, this type of postgraduation assessment has failed to pen¬ 
etrate widely below the institutional level, and the data gathered through 
institutional assessment rarely involve the kind of specificity that programs 
need to understand why a program or an element of a program is succeed¬ 
ing or failing and how it might be improved. By adopting postgraduate as¬ 
sessment, academic programs can use the experiences and perspectives of 
former students to improve the teaching and learning of current and future 
students and retool programmatic design. Moreover, proactively adopting 
such measures prepares the program for future institutional and governing 
body policy revisions, such as increased or more nuanced accountability 
demands, that are most likely inevitable (See for example, Mangan, 2011, 
and Patton, 2012). 
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We found that collecting data on student learning solely while students 
were in the program had left us with a partial view of our impact on their 
academic and professional development. By querying graduated students 
we were able to complete the picture. The perspective granted by real-world 
experience, when combined with the other evaluation and assessment mea¬ 
sures we had implemented during their time at the university, provided us 
with multilayered insights into where the program was succeeding and 
where it needed to improve. 

A Practical Guide to Postgraduation Evaluation and Assessment 

Any interdisciplinary program can successfully design and administer a 
postgraduation evaluation and assessment tool. The project involves some 
thoughtful consideration in the design phase, research to compile a subject 
pool, a straightforward collection of data, and an analysis stage that can 
range from the running of basic statistical tests to more intricate variance 
analysis depending on the complexity of the information desired. Using our 
experience at West Virginia University as an example, the guide that follows 
is intended to assist other interdisciplinary studies programs in navigating 
the difficulties in developing a postgraduation evaluation and assessment 
tool and carrying out a plan for collecting data on a large scale. 

Know Your Mission 

Understanding your program’s mission is the first phase of programmat¬ 
ic postgraduation evaluation and assessment. Our program was founded 
in 2006 and requires interdisciplinary students to complete the universi¬ 
ty’s undergraduate general education requirements, three minors selected 
from over 100 available choices, and only two interdisciplinary courses, 
an introduction to interdisciplinarity and an integration-focused, writing¬ 
intensive capstone. The Dean’s office set our programmatic mission as en¬ 
couraging undergraduate retention and satisfying students’ demands for 
flexibility by providing them with an avenue to formulate individualized 
plans of study. As of spring semester 2012, the program has grown to just 
over 600 undergraduate students. The large number of enrolled students, 
limited student-contact hours, and disciplinary variation were critical ele¬ 
ments to consider when implementing our mission of retention and indi¬ 
vidualized studies. 

In addition to addressing these pragmatic concerns, we felt it imperative 
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to integrate a strong academic understanding of interdisciplinarity into our 
curriculum. We reviewed much of the seminal work in IDS to develop our 
own understanding of interdisciplinarity, but found the concept of “deep 
learning” as presented by Stowe and Eder (2002) to be a particularly use¬ 
ful concept in developing this portion of our mission. The authors define 
“deep learning” as “the learning possessed by experts in a field,... usually 
acquired directly through instruction and indirectly through experience” 
(p. 82) and argue convincingly that interdisciplinary learning is inherently 
“deep learning.” Stowe and Eder state that to achieve “deep learning” in 
an academic environment at least four factors must be present: 

1. Students must experience a pedagogy beyond “standard” passive 
lecture. 

2. Students must face a significant problem for which their own dis¬ 
cipline and its way of knowing is a necessary but not a sufficient 
contributor to the solution. 

3. Students must see the need for acquiring relevant and important 
facts outside their own major disciplines and inserting them into 
new contexts. 

4. Students must practice acquiring those facts and manipulating 
them in those contexts under faculty supervision. 

(Stowe & Eder, 2002, p. 82) 

We agreed, as faculty and as interdisciplinarians, that combining these 
ideals with our more pragmatic charges had the potential to foster the learn¬ 
ing environment we hoped to create for our students. Our programmatic 
mission and the learning outcomes to follow were developed with this con¬ 
ception of interdisciplinary “deep learning” in mind. 

Set Your Learning Outcome Goals 

Within the boundaries set by our mission, we could now develop the 
curriculum that best served our students. By referencing the IDS liter¬ 
ature, we learned of the value and difficulty in integrating disciplinary 
foundations (see Boix Mansilla, 2005; Repko, 2007, 2008; and Newell, 
1994, 2001, 2006). Our first experience with undergraduate IDS students 
immediately raised our awareness of the importance and challenge of 
overcoming a lack of name recognition for our program and demonstrat¬ 
ing the employability of students with our degree. To surmount these ob¬ 


Interdisciplinary Studies and the Real World 


stacles, we developed a set of programmatic learning outcome goals that 
continue to serve as our core principles for maintaining a strong under¬ 
graduate program: 

1. Knowledge 

A. Broad-based knowledge of three discrete areas of study. 

B. Understanding of the synergistic advantage of interdisciplin¬ 
ary curriculum. 

2. Skills 

A. Ability to think critically in each of three disciplines. 

B. Ability to partition and interpret information or events using 
the most appropriate discipline’s toolset or by combining 
disciplines’ toolsets. 

C. Ability to integrate knowledge and modes of thinking drawn 
from multiple disciplines. 

D. Ability to write a professional resume, conduct a job inter¬ 
view, and apply to graduate school. 

E. Ability to research and write a research paper. 

F. Ability to create and present a quality visual presentation. 

3. Attitudes 

A. Positive attitude towards civic action and engagement. 

B. Familiarity with poverty in West Virginia/America. 

4. Integrative Learning 

A. Ability to explain interdisciplinary studies and its advantages 
to others. 

B. Ability to synthesize and integrate academic knowledge to 
confront real-world problems. 

Develop Your Postgraduation Evaluation and Assessment Tool 

Testing the imperatives of our programmatic mission and our desired 
learning outcomes constituted the foundation for our programmatic evalu¬ 
ation and assessment plan. This article is concerned primarily with post¬ 
graduation evaluation and assessment, but we would be remiss if we did 
not make it clear that we view postgraduation evaluation and assessment 
as only an important piece of a larger evaluation and assessment plan. In 
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addition to examining the evaluative data gathered at the institutional level, 
we actively carry out direct and indirect, quantitative and qualitative evalu¬ 
ation and assessment at the course and program level. For example, at the 
programmatic level, we have developed new-student and graduating-student 
surveys, which incorporate open-ended responses to provide data on what 
our students have learned relative to our programmatic learning outcome 
goals. At the course level, we engage in authentic assessment through the 
use of mbrics and peer review on individual assignments to determine 
whether students are developing the specific skills we had hoped for when 
we created the assignments. 

It was while we were calibrating these programmatic evaluations and as¬ 
sessments that we realized we lacked our graduates’ perspectives on our 
program. It was to correct this oversight that we first began developing a 
postgraduation evaluation and assessment tool. Ultimately, we chose to cre¬ 
ate a 50-question survey with 44 closed-ended questions or prompts and 
six open-ended questions. The six open-ended questions allowed the re¬ 
spondents to offer their perspective on the topics raised in the closed-end¬ 
ed prompts. The prompts were broken up into four distinct informational 
groupings: general, programmatic, capstone-specific, and specific skill de¬ 
velopment. These queries provided us with the baseline demographic infor¬ 
mation necessary for conducting more nuanced and sophisticated analysis 
of our learning outcome goals. The programmatic, capstone-specific, and 
specific skill development prompts also each addressed some aspect of our 
learning outcome goals. For example, in our capstone-specific section we 
asked the participants to respond to the following prompt: 

The research paper assignment in the Capstone course helped me to 
produce an interdisciplinary’ understanding of a complex problem or 
intellectual question. 

Strongly Disagree Disagree Unsure Agree Strongly Agree 

The answers we received on this question allowed us to assess issues re¬ 
lated to learning outcome goals 1A, IB, 2A, 2B, 2C, 2D, and 4B. When we 
combined the responses with other prompts that also address these learning 
outcome goals, and then cross-referenced the data with more direct in-class 
assessment efforts such as assignment mbrics and peer reviews, we devel¬ 
oped a much clearer picture of how well we had served our students. 
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Query Your Foundation 

After constructing our postgraduation evaluation and assessment tool, we 
moved on to the data collecting phase. This stage of the project can be dif¬ 
ficult as the challenge of contacting potential respondents can form a con¬ 
siderable obstacle to implementing postgraduate assessment. We pursued 
two avenues with some success, and we plan to implement two additional 
options in the future that we hope will increase our respondent pool and our 
response yield. 

We first requested a database of students who had graduated from the 
program in the past 18 months and their student email addresses. At West 
Virginia University, students retain access to their email for approximately 
one year after graduation and occasionally a little longer. At present, the 
University’s technical staff has resisted calls to provide students with per¬ 
manent access to their University email addresses. In universities where 
students retain their undergraduate emails permanently, the data collection 
phase would most likely be rendered considerably less difficult. 

We supplemented the email addresses of recent graduates with a request 
to our University’s alumni foundation. To our delight, the West Virginia Uni¬ 
versity Foundation provided us with hundreds of email addresses. Flowever, 
unless the students had responded to earlier requests from the Foundation, 
the contact information was often woefully outdated. Still, contacts retrieved 
from these two sources formed the bulk of our pool. Ultimately, we ended 
up with a respondent pool of just under 500 alumni, 102 of whom actually 
completed our survey. 

A third option, which we have not yet pursued, is to mail requests for an 
email contact to the permanent mailing address on file with the Founda¬ 
tion. Previous experience with physical mailings suggests that this avenue 
will not be particularly cost-effective, requiring postage, or effective, as it 
demands active participation from the letter recipient, but it may be a useful 
way to supplement the first two avenues. 

The final means by which we hope to increase our participant pool is to 
adapt social media to the problem. There are a number of ways to try to 
reach out to alumni through social media outlets. The two easiest, and most 
likely to be successful, are to create a Facebook profile and a Twitter feed 
that send out periodic messages to alumni. Through these sites we can ask 
for alumni email addresses or send a direct link to our survey instrument. 
Given the relative youth of our alumni base, this avenue is likely to be more 
successful than physical mailing, but it does still demand active participa- 
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tion from graduates. Of course, so does the act of answering a survey. It is 
our hope that if our alumni base is willing to communicate with us through 
Facebook and Twitter, they will be willing to engage us through our evalua¬ 
tion and assessment tool as well. 

The Internet and Bribery 

Email addresses in hand, we sent each of our contacts a link to our survey 
along with a plea to help us improve our program and then repeated the 
process several times over a few months. We also sent each recipient a PDF 
copy of our alumni newsletter containing a similar plea. We supplemented 
our appeals to our graduates’ good natures by entering survey participants 
in a small sweepstakes. The state of West Virginia fortunately allows state 
funds to be used for “promotional materials,” and by filling out the survey 
each respondent was given a chance to win a $50 gift card to the University 
bookstore. We provided three gift cards at a total cost of $150. 

Analyze Your Data 

After creating an evaluation and assessment tool, determining our re¬ 
spondent pool, and collecting as many responses as possible, it was time to 
analyze the data. There is a wide array of analysis measures available, and 
it was important to determine which measures made the most sense consid¬ 
ering our mission and learning outcome goals. After exploring some of the 
available tools, we were able to discover what worked for our needs. 

First, we let Survey Monkey do much of the work for us. This web-based 
software provides users with data on how participants responded to quan¬ 
titative questions and can also create a wide variety of easy-to-read graphs 
and charts. These data can then be used to run basic descriptive statistics 
such as mean, median, and mode, providing the basic information neces¬ 
sary for yearly programmatic assessments reports and five-year accredita¬ 
tion reports. Survey Monkey is easy to use and reasonably priced at several 
different subscription levels. 

The software also makes possible conducting more thorough analysis, 
assessing learning outcomes or curricular changes longitudinally, and de¬ 
termining the impact of multiple issues/criteria on learning outcomes. Sur¬ 
vey Monkey facilitates the process by allowing the user to create cross-tabs 
where multiple questions are assessed in combination. To gain more insight, 
it was useful, but not necessary, to run software like Excel and SPSS. These 
programs allowed us to run analyses like T-Tests, which check for a statisti¬ 
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cally significant difference between two means. We found SPSS to be the 
more sophisticated of the two as it performs many different types of statisti¬ 
cal analysis. 

To provide a brief example, suppose we implemented an experiential 
service learning initiative at the start of 2011 to help students engage the 
world from multiple perspectives. We know what year students graduated 
by including the question “In what year did you graduate?” and we can test 
students’ responses to the initiative by asking, “Did our program help you 
to develop the ability to engage the world from different points of view?” 
We can then evaluate graduated student responses to the latter question 
based on the year they completed the program. Survey Monkey provides 
the basic cross-tabbed statistics, which we can then download to a program 
like Excel or SPSS to run a T-Test to check for a statistically significant dif¬ 
ference between the two years. By executing other analyses of this sort we 
can assess the value of curricular changes in some detail, fostering program 
accountability, and improving our ability to accomplish our programmatic 
mission. 

Our last piece of advice addresses open-ended questions. To demonstrate 
accountability and pass accreditation, quantitative analysis speaks volumes. 
However, we discovered that some audiences also appreciate when our as¬ 
sessment reports are punctuated by “testimonials.” During the presentation 
of data analysis, it can be beneficial to present relevant stories or direct 
quotes from your respondents to drive home your point. For some audi¬ 
ences it is easier to appreciate a program’s value by hearing directly from 
the students. 

The guide, to this point, has shown evaluation and assessment from their 
inception through analysis. Analysis, however, is nothing without applica¬ 
tion. The final section of this article demonstrates the applicability of post¬ 
graduation research to evaluating and assessing program mission and learn¬ 
ing outcome goals through an example derived from our research tool. 

A Practical Application of Our Postgraduation 
Evaluation and Assessment 

Upon retrieving our data, we were struck immediately by the relevance of 
some of our more basic queries to the University’s programmatic review and 
evaluation process. To offer just one example, our postgraduate assessment 
tool concluded with the following prompt: 
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Are you satisfied with your degree choice? 
Yes No 


The preponderance of positive responses provides persuasive evidence for 
the success of the program, but this information would have been unavail¬ 
able to us if we had not begun to collect postgraduate evaluative data. Until 
we queried the students after graduation, we had generally relied on anec¬ 
dotal evidence to convey the real-world worth of our degree program. After 
administrating the survey, we possessed statistical evidence that, even in a 
down economy, choosing an interdisciplinary curriculum was perceived as a 
perspicacious choice by the great majority of our graduates. 

Such demonstrations of programmatic effectiveness are exceedingly valu¬ 
able for successfully negotiating both external and internal programmatic 
reviews, but they do not tell the complete story. To take full advantage of the 
potential of postgraduate program assessment, it is necessary to do more than 
ask evaluative questions. To illustrate this point, it may be useful to provide a 
thorough example examining one aspect of our postgraduate survey through 
the lens of a single issue, in this case an undergraduate research paper. 

One of our programmatic learning outcome goals expresses our desire 
for students to understand the synergistic advantage of an interdisciplinary 
curriculum. Although we provide a number of assignments with this learn¬ 
ing goal in mind, the primary assignment in our Interdisciplinary Studies 
Capstone course is for students to write a research paper designed to foster 
disciplinary critical evaluation and interdisciplinary thinking. 

Our classroom assessment of the assignment returned mixed results. Fac¬ 
ulty rubrics revealed that many students had only limited success thinking 
in an integrative manner. Moreover, when we asked students what they felt 
about the research paper at the end of the course, the majority felt that the 
paper assignment was not one of the more useful aspects of the class, rank¬ 
ing it dead last in value out of the 10 course assignments. These results 
encouraged us to consider eliminating the research paper assignment. After 
some discussion, we decided to proceed with more analysis first, examining 
the question of integrative thinking in our postgraduate survey. Perhaps test¬ 
ing the value of the analytical essay for fostering integrative thinking in our 
postgraduate survey would expose new avenues for positive change. 

We asked our alumni to respond to the following prompt: 





Note: This chart was generated by Survey Monkey (surveymonkey.com) 
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The MDS program helped me to integrate knowledge and modes of 
thinking drawn from multiple disciplines /minors. 

A total of 101 graduates answered the question, and Survey Monkey ex¬ 
ported this chart on the preceding page. 

Our graduates overwhelmingly agreed that they had learned how to inte¬ 
grate disciplinary knowledge, but where had they learned these skills? Our 
faculty’s attempts to teach integrated thinking largely occurred in the In¬ 
terdisciplinary Studies Capstone course, so we refined the assessment with 
another prompt: 

The Capstone coursework helped me to synthesize knowledge from 
multiple disciplines/minors. 

(See Table 3 on the following pages.) 

Although the majority of students continued to accept the premise, the num¬ 
bers were less consistently positive than when they commented on their aca¬ 
demic experience holistically. This seemed to suggest that our Capstone as a 
whole was not working as well as it could, but we still needed to know more 
about the research paper’s specific role in fostering integrative thinking: 

The research paper assignment in the Capstone course helped me to 
produce an interdisciplinary understanding of a complex problem or 
intellectual question. 

(See Table 4 on the following pages.) 

This finding surprised us. In retrospect, graduates felt that the paper assign¬ 
ment had done a better job preparing them for interdisciplinary work than 
either faculty rubrics or immediate post-course assessment would have sug¬ 
gested. In fact, students found the paper more useful than the class as a 
whole. Our survey had provided a new perspective with which to consider 
the Interdisciplinary Studies Capstone generally and the research paper spe¬ 
cifically. 

Postgraduate assessment thereby changed our course of action in the pro¬ 
gram. Rather than deciding that an interdisciplinary research project was 
too complicated for our undergraduate students, we decided to act on the 
opinion of our students who had seen the paper as much more useful in 
retrospect than at the conclusion of the course. We rewrote and extended the 




94 


Benjamin Brooks & Evan Widders 



Table 3 


Table 4 


Interdisciplinary Studies and the Real World 


95 









96 


97 


Benjamin Brooks & Evan Widders 


assignment, provided additional course time for planning the paper, limited 
the potential topics, and changed the focus from research to disciplinary 
analysis. We hope that future assessment will prove that these changes have 
provided measurable improvements towards achieving our learning goals in 
the Capstone and with the paper project. If so, we can highlight this effort in 
our yearly assessment report and our five-year review as an example of our 
program’s accountability to student learning outcomes and of our willing¬ 
ness to modify our curriculum to better achieve our programmatic mission. 

Conclusion 

We hope the process described in this paper will serve as a persuasive ex¬ 
ample for initiating regular programmatic postgraduation evaluation and as¬ 
sessment. Admittedly, beginning postgraduation evaluation and assessment 
can be an intimidating prospect in a down economy. When we began, it 
certainly seemed possible that our students would blame their academic ex¬ 
perience for difficulties they might have experienced in the job market and 
that their responses to our queries could reflect negatively on our program 
or even jeopardize our future. Once we embarked on the process, however, 
we found that the experience was almost wholly positive. Negative feedback 
was largely focused on specific areas and was useful for programmatic im¬ 
provement, while the majority of our respondents were enthusiastic about 
their experience in the program. We consider these results to be a powerful 
endorsement of the tenets of interdisciplinary studies and a convincing ex¬ 
ample of the applicability of IDS programs to success in the “real world.” 

Biographical Note: Assistant Professor Evan Widders is Director of the Multidisci¬ 
plinary Studies Program at West Virginia University. Assistant Professor Benjamin' 
Brooks is a joint faculty member in the Multidisciplinary Studies and Leadership 
Studies programs at WVU. The Multidisciplinary Studies Program is the second 
largest undergraduate program in the University. They can be reached, respectively, 
at evan.widders@mail.wvu.edu and benjamin.brooks@mail.wvu.edu. 
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